Multi-sensor high dynamic range imaging

ABSTRACT

High dynamic range imaging includes first image data captured using a first image sensor and a first exposure time, using a rolling shutter such that different lines within the first image data are captured at different times. Two or more instances of second image data are captured using second image sensors and second exposure times that are shorter than the first exposure time. The second image data are captured using a rolling shutter, and overlap at least in part with the first image data. A line of the first image data has a corresponding line in each instance of second image data, and the corresponding instances of second image data are captured at different second capture times. For a line in the first image data, the corresponding line from the instance of second image data is selected to be merged with the line in the first image data.

BACKGROUND

The present invention relates to image sensors, and more specifically toimaging devices with arrays of image sensors that may be used to produceHigh Dynamic Range (HDR) images.

Image sensors are commonly used in electronic devices such as cellulartelephones, cameras, and computers to capture images. In a typicalarrangement, an electronic device is provided with a single image sensorand a single corresponding lens. In certain applications, such as whenacquiring still or video images of a scene with a large range of lightintensities, it may be desirable to capture HDR images. In HDR images,highlight and shadow detail can be retained that would otherwise be lostin a conventional image.

A common type of image sensor is a Complementary Metal OxideSemiconductor (CMOS) image sensor. In a CMOS sensor, photoelectricconversion elements of pixels perform photoelectric conversion foraccumulating charges according to incident light amounts, and outputelectrical signal corresponding to the accumulated charges.

A CMOS sensor typically has an electronic shutter function. Theelectronic shutter function starts exposure by resetting photoelectricconversion elements of pixels, and ends exposure by reading out chargesaccumulated on the photoelectric conversion elements. In this manner,since the start and end of exposure are controlled by only the functionof the image sensor, an exposure time from a low-speed shutter to a highspeed shutter can be accurately controlled.

HDR typically works by merging a short exposure and a long exposure ofthe same scene, making the CMOS sensor suitable for an HDR image capturesetup. Sometimes, more than two exposures can be involved. Sincemultiple exposures are captured by the same sensor, the exposures arecaptured at different times, which causes temporal problems in terms ofmotion blur.

One way to address this problem involves placing two image sensors veryclose to each other (similar to the setup of stereo camera, althoughwith much shorter distance between the sensors as no disparity isrequired). In such a setup, one sensor is configured for long exposure,and the other sensor is configured for short exposure. The long exposureimage and the short exposure image are then merged to create an HDRimage. The captures are synchronized in time so that the merge into anHDR image can work more smoothly.

A further problem is that CMOS sensors generally use a rolling shutteroperation (sometimes often referred to as a “focal plane shutter”operation). With the rolling shutter operation, charges on pixels arereset by sequentially scanning a plurality of two dimensionally arrangedpixels for each line. Then, after an elapse of a predetermined exposuretime, the pixels are sequentially scanned for each pixel to read outaccumulated charges and to output signals. This sequential scanningcauses different rows of the sensor to be read out at different pointsin time. Because of this, when merging a row in the long exposure imagewith a corresponding row in the short exposure image, problems withmotion blur will still occur due to the rows being read out at differentpoints in time. For at least these reasons there is a need for bettermethods of creating HDR images.

SUMMARY

According to a first aspect, a method, in a computer system, for highdynamic range imaging, includes:

-   -   capturing first image data using a first image sensor and a        first exposure time, wherein the first image data is captured        using a rolling shutter such that different lines within the        first image data are captured at different first capture times;    -   capturing two or more instances of second image data using one        or more second image sensors and one or more second exposure        times that are shorter than the first exposure time, wherein:        -   the two or more instances of second image data are captured            using a rolling shutter such that different lines within            each instance of second image data are captured at different            second capture times,        -   the two or more instances of second image data overlap at            least in part with the first image data,        -   a line of the first image data has a corresponding line in            each instance of second image data, the corresponding lines            in the different instances of second image data being            captured at different second capture times; and    -   for a line in the first image data, selecting the corresponding        line from the instance of second image data whose second capture        time is closest to the first capture time, to be merged with the        line in the first image data to generate a high dynamic range        image.

This provides a way of improving techniques for HDR imaging, inparticular with respect to how long- and short exposure lines are mergedwhen a rolling shutter is used, such that artefacts from HDR are reducedand an improved HDR image is obtained.

According to one embodiment the second exposure times are different fordifferent instances of second image data. That is, certain shortexposures may be very short and others may be slightly longer, whichoffers flexibility in terms of which lines to merge into the HDR imagefrom the short and long exposures.

According to one embodiment, the two or more instances of second imagedata are captured using two or more second image sensors, wherein atleast one of the second exposure time and the second capture time differbetween the two or more second image sensors. Having a setup withmultiple short exposure sensors can allow for the two sensors, forexample, to record images where the exposure time differs between thesensors, or where different sensors record different wavelengths in theelectromagnetic spectrum. This can provide even further flexibility inselecting which lines to merge into the HDR image.

According to one embodiment, instances of second image data captured bythe two or more image sensors overlap at least in part. That is, thecapturing of short exposure images can be configured so that there is no“gap” between the capture of short exposure images. This makes itpossible to further reduce the time difference between how lines arecaptured and read out when merging the long and short exposure images.

According to one embodiment, the first and second capture times, aredetermined with respect to one or more of: a reset of a line and a readof a line This allows flexibility in how the “reference point” isselected when determining which lines to merge. Sometimes it may beuseful to have the reset of a line as the reference point. Other times,it may be useful to have the read of a line as the reference point, andother times, any specific point in between the read and reset can beused. Again, this provides the flexibility to select the “best” linefrom each exposure to be combined into an HDR image and to reduce imageartifacts.

According to one embodiment, the first image sensor and the one or moresecond image sensors have different line times. The different line timesmay be beneficial in that the sensors can be read out with differentspeeds. That is, more short exposures can easily be fitted within a longexposure.

According to one embodiment, two corresponding lines may be selectedfrom the two instances of second image data whose second capture timesare closest to the first capture time, to be merged with the line in thefirst image data when generating a high dynamic range image. That is,the selection can involve the corresponding line from each of two ormore short exposures. There may be circumstances under which the closestshort exposure is not necessarily the best one for HDR purposes. In suchscenarios, a corresponding line from a different short exposure imagescan also be selected to compensate for this drawback and be usedtogether with the closest short image exposure.

According to one embodiment, a weight can be assigned to each selectedline among the selected two corresponding lines, to be used in thegeneration of the high dynamic range image. That is, it is possible toselect, for example, that the corresponding line from the closest shortexposure is more “important” and should be assigned more weight than acorresponding line from a short exposure that is further away in time.

According to one embodiment, at least one of: the first exposure times,the first capture times, the second exposure times, and the secondcapture times are selected in response to a user input. That is, a usermay configure exposure times and capture times for the different sensorsto the particular situation at hand such that the best possible HDRimage can be generated.

According to one embodiment, the selecting of a corresponding line fromthe instance of second image data can be done for each line in the firstimage data. Making such a selection for each line in the first imagedata ensures that the best selection of lines is made for the entireimage, and thus provides an HDR image having high quality throughout theentire image.

According to one embodiment, input can be received regarding a region ofinterest in the first image data, wherein, for each line in the firstimage data intersecting the region of interest, a corresponding linefrom a same instance of second image data is selected. That is, whilethe basic rule is to select a line which is captured closest in time,this rule does not need to be applied to every line in the image.Specifically, a user may be interested in specifying a region within animage where there should not be any discontinuities, as could be thecase when lines from different instances of second image data are used.Therefore, for such regions, it may be better to select a line from thesame instance of second image data. The proper instance of second imagedata for a line in the first image data that overlaps the specifiedregion can be identified using the basic rule of temporal proximity asdescribed above. Then, for the other lines in the first image data thatalso overlap the specified region, the corresponding line can beselected from that same instance of second image data, even if therehappens to be another instance of second image data that has a capturetime closer to the first capture time. Using this method may avoid or atleast significantly reduce the discontinuities in the resulting HDRimage.

According to a second aspect, a system for high dynamic range imagingincludes a memory and a processor. The memory contains instructions thatwhen executed by the processor causes the processor to perform a methodthat includes:

-   -   capturing first image data using a first image sensor and a        first exposure time, wherein the first image data is captured        using a rolling shutter such that different lines within the        first image data are captured at different first capture times;    -   capturing two or more instances of second image data using one        or more second image sensors and one or more second exposure        times that are shorter than the first exposure time, wherein:        -   the two or more instances of second image data are captured            using a rolling shutter such that different lines within            each instance of second image data are captured at different            second capture times,        -   the two or more instances of second image data overlap at            least in part with the first image data,        -   a line of the first image data has a corresponding line in            each instance of second image data, the corresponding lines            in the different instances of second image data being            captured at different second capture times; and    -   for a line in the first image data, selecting the corresponding        line from the instance of second image data whose second capture        time is closest to the first capture time, to be merged with the        line in the first image data to generate a high dynamic range        image.

The system advantages correspond to those of the method and may bevaried similarly.

According to a third aspect, a thermal camera includes a system asdescribed above, for high dynamic range imaging. The advantages of thecamera correspond to those of the system and may be varied similarly.

According to a fourth aspect, a computer program for high dynamic rangeimaging contains instructions corresponding to the steps of:

-   -   capturing first image data using a first image sensor and a        first exposure time, wherein the first image data is captured        using a rolling shutter such that different lines within the        first image data are captured at different first capture times;    -   capturing two or more instances of second image data using one        or more second image sensors and one or more second exposure        times that are shorter than the first exposure time, wherein:        -   the two or more instances of second image data are captured            using a rolling shutter such that different lines within            each instance of second image data are captured at different            second capture times,        -   the two or more instances of second image data overlap at            least in part with the first image data,        -   a line of the first image data has a corresponding line in            each instance of second image data, the corresponding lines            in the different instances of second image data being            captured at different second capture times; and    -   for a line in the first image data, selecting the corresponding        line from the instance of second image data whose second capture        time is closest to the first capture time, to be merged with the        line in the first image data to generate a high dynamic range        image.

The computer program involves advantages corresponding to those of themethod and may be varied similarly.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features and advantages willbe apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are schematic diagrams showing a camera 100 in which thesystem and methods in accordance with various embodiments can beimplemented.

FIG. 2 is a flowchart depicting a method selecting image data to bemerged into an HDR image, in accordance with one embodiment.

FIG. 3 is a schematic diagram showing the relation between a longexposure image and short exposure images captured over time, inaccordance with one embodiment.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

As was described above, it would be beneficial to provide improvedtechniques for HDR imaging, in particular with respect to how long andshort exposure lines are merged when a rolling shutter is used, suchthat motion blur and other artefacts are reduced. At a general level,the various embodiments work as follows.

Rather than using a single image sensor to capture long exposure andshort exposure images, respectively, two or more sensors are placed inclose proximity. One of the sensors is used to capture a long exposureimage, and one or more of the other sensors is used to capture severalshort exposure images that at least in part overlap with the exposuretime of the long exposure image. The sensors use a rolling shutter. Eachline in the sensor having the long exposure image has a spatiallycorresponding row in each of the sensors having a short exposure image.

When the images have been captured, lines are selected from the long andshort exposure images for merging into an HDR image. In particular, foreach long exposure line, a corresponding line is selected from the shortexposure in which the line has the closest capture time with respect tothe capture time of the long exposure line. In some implementations morethan one corresponding line can be selected from different shortexposure images, as will be described in further detail below. Thisselection of lines produces fewer motion artefacts compared toconventional techniques. The selected lines are sent to an HDR imagegenerator that generates the HDR image, using conventional techniquesthat are well known to those having ordinary skill in the art.

Various embodiments will now be described in further detail and by wayof example with reference to the drawings. FIGS. 1A and 1B showschematic views of a camera 100 in which various embodiments can beimplemented. As can be seen in FIGS. 1A and 1B, the camera has a lens102 through which incident light passes and reaches a semi-transparentmirror 104. The mirror 104 acts as a beam splitter and divides theincident light up into components, which each reaches a sensor.

In FIG. 1A, there are two sensors, 106 and 108, respectively. In FIG.1B, there are three sensors, 106, 108 and 110, respectively. Each ofthese sensors record the same image. The sensors shown in FIGS. 1A and1B are conventional CMOS sensors that use a rolling shutter operation asdescribed above.

It should be noted that the mirror 104 does not necessarily have to be amirror, but can be any optical component that has the ability ofdividing incident light into multiple components, such as a prism, forexample. It should also be noted that as an alternative to using a beamsplitter (i.e., mirror, prism, etc.), the two image sensors mayalternatively be arranged side-by-side at a close distance to eachother, similar to the setup of stereo camera, although with much shorterdistance between the sensors as no disparity is required. It shouldfurther be noted that while only two and three sensors are shown inFIGS. 1A and 1B, respectively, this is done for illustration purposesonly, and there may also be implementations that use a larger number ofsensors.

Preferably, the sensors 106, 108, 110, have similar properties in termsof resolution, pixel characteristics, etc., in order to reduce artefactswhen combining images from two or more of the sensors. However, this isnot a requirement, and there may be situations in which some otherbeneficial properties of the sensors 106, 108 and 110 may outweigh thedrawbacks of possible image and motion artefacts. Such design choicesfall well within the capabilities of a person having ordinary skill inthe art.

FIG. 2 is a flowchart showing a method 200 for selecting lines to beused in HDR imaging, in accordance with one embodiment. As can be seenin FIG. 2, the method 200 starts by capturing first image data andsecond image data, step 202, with sensors 106 and 108, respectively, asillustrated in FIG. 1A. Typically, sensor 106 has a long exposure timeand sensor 108 has a short exposure time, such that multiple images arecaptured by sensor 108 while sensor 106 captures a single image. Itshould be noted that there does not need to be an exact overlap betweenwhen the long and short exposures, respectively, start. It is typicallysufficient that there are two or more shorter exposures within one longexposure, and the exact placement in time for these shorter exposuresmay vary.

FIG. 3 provides a conceptual illustration 300 of how the two sensors 106and 108 capture short exposure and long exposure images, respectively,of the same scene. At the bottom of FIG. 3 is a timeline 302. At the topis a representation of the long exposure, L, captured by sensor 106 andbelow that are representations of two short exposures S1 and S2,respectively, captured by sensor 108. As can be seen in FIG. 3, theshort exposures S1 and S2 are captured during the capture of the longexposure L. However, it should be noted that the capture of S1 and S2does not need to be contained within the capturing time of L. There maybe situations when the capture of S1 starts before the capture of Lstarts, for example, and there may be situations when the capture of S2finishes after the capture of L finishes.

When a rolling shutter is used, different rows of a sensor are read outat different points in time. This is illustrated by horizontal lines 304a-c and 306 a-c in FIG. 3. For example, for the long exposure L, row 304a is read before row 306 b. The corresponding situation applies to rows304 b and 306 b of exposure S1 and rows 304 c and 306 c of exposure S2.FIG. 3 also illustrates how sensor 108 has a shorter readout timecompared to sensor 106, by having a steeper “slope.”

It can also be seen in FIG. 3 how a row in the long exposure L has acorresponding row in each of the short exposures S1 and S2,respectively. In particular, in the example illustrated in FIG. 3, row304 a in the long exposure L has corresponding rows 304 b in shortexposure S1 and 304 c in short exposure S2. Row 306 a in the longexposure L has corresponding rows 306 b in short exposure S1 and 306 cin short exposure S2.

When the HDR image is generated, a row in the long exposure L is mergedwith a spatially corresponding row from one of the short exposures S1,S2. Returning now to FIG. 2, the method continues to select, for eachline in the long exposure L, one or more corresponding lines from one ormore of the short exposures S1 and S2 to be merged with the line in thelong exposure L, step 204. This selection should be done in a way thatmitigates the negative effects of using a rolling shutter to the largestextent possible, for example to eliminate problems with HDR merge motionartefacts due to the rows being read out at different points in time.

In one implementation, this is addressed by selecting the correspondingrow from the short exposure S1, S2, that is exposed closest in time tothe row of the long exposure L. To measure this “closeness in time,” acapture time may be assigned to each row, for example, the capture timemay be the point in time that is located half way between a reset and aread of the row. For example, in FIG. 3, time tL1 represents the halfway time point for line 304 a of the long exposure L and would thus bedesignated the capture time for line 304 a. Correspondingly, time tS1represents the half way time point for line 304 b of the short exposureS1, and time tS2 represents the half way time point for line 304 c ofthe short exposure S2. Thus, tS1 and tS2 would be the capture times forlines 304 b and 304 c, respectively. However, as the skilled personrealizes this half way time point is merely one choice and otheralternative capture times may also be considered, such as letting thecapture time equal the read-out time or any other time point between areset and a read of a line. Such design choices can easily be made bythose having ordinary skill in the art.

As can be seen in FIG. 3, for the long exposure line 304 a with capturetime tL1, the closest corresponding line in S1 and S2, is line 304 b inshort exposure S1, with capture time tS1. Thus, for purposes ofperforming HDR, line 304 a in the long exposure L would be selected tobe merged with the corresponding line 304 b in the first short exposureS1. On the other hand, line 306 a in the long exposure L would insteadbe selected to be merged with the corresponding line 306 c of the secondshort exposure S2. Ultimately, this means that different rows of pixelscan be fetched from different instances of short exposures to compensatefor the rolling shutter effect when composing the HDR image.

Returning to FIG. 2 again, the selected rows from the differentexposures are sent to an HDR image generator, which generates an HDRimage using techniques that are well known to those having ordinaryskill in the art, step 204. This ends method 200.

It should be noted that in the above example, the line time (that is,the time between the reset of two consecutive rows on a sensor) isdifferent for the long exposure sensor 106 and the short exposure sensor108. Specifically, the line time for the long exposure sensor 106 islonger than that of the short exposure sensor 108. Having sensors withsuch properties is expected to improve the results and yield better timesynchronized matches between rows of the long and the short exposures.However, it should be noted that it is not a requisite for theembodiments to work. There are still benefits of using this setup, evenwhen the line times are identical for all sensors, as the sensorscapture the exact, or at the very least nearly the exact, same scene andthe images from the sensors are closer in time (compared to theconventional case when a single sensor is used to alternate short andlong exposures).

Further, in some implementations, to reduce discontinuities due tochoosing different short exposures for different rows, the shortexposures may be blended. Typically, when blending occurs, lines can beassigned different weights. For example, the line that is closest intime, and that would normally be chosen as described above, can beassigned a weight that is higher than a corresponding line from adifferent short exposure that is further away in time. For example, withreference again to FIG. 2, instead of selecting line 304 b to be mergedwith line 304 a, as is done in a typical embodiment, it may prove usefulto select not only line 304 b but also line 304 c to be merged with line304 a in order to reduce discontinuities in the resulting HDR image.However, as line 304 c is located further away in time compared to line304 b with respect to line 304 a, line 304 c is given less weight. Forexample, a blended line can be created that where each pixel value is acombination of 80% of the pixel value in line 304 b, and 20% of thepixel value in line 304 c. This blended line can then be sent to the HDRimage generator along with the long exposure line 304 a. Many variationscan be envisioned by the skilled artisan.

It should also be noted that while the example above refers to a setupusing two sensors, such as the one shown in FIG. 1A, there may be evenmore advantages with using more sensors, such as the setup shown in FIG.1B. In that configuration, sensors 108 and 110 can both be shortexposure sensors, and can be timed such that the exposures of sensor 108occur in between the exposures of sensor 110, or that the shortexposures from sensors 108 and 110 even overlap in part. This provides alarger number of lines from which a “best match” can be selected (or alarger number of lines that can be weighed and blended in variousconfigurations). Exactly how to do this will depend on the particularsituation at hand and is considered a matter of design choice that canbe determined by a person having ordinary skill in the art.

In some implementations, one of the sensors, such as sensor 110, forexample, could have a different exposure time compared to sensors 106and 108. For example, there may be a short exposure sensor 108, a mediumexposure sensor 110, and a long exposure sensor 106. This can yieldadditional benefits, such as allowing capture of the image with betterquality throughout a wider range of luminance levels, as compared towhat would be possible when merging only a long and a short exposure.

As will be appreciated by one skilled in the art, aspects may beembodied as a system, method or computer program product. Accordingly,some aspects may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “circuit,”“module” or “system.” Furthermore, other aspects may take the form of acomputer program product embodied in one or more computer readablemediums having computer readable program code embodied thereon.

Any combination of one or more computer readable mediums may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electromagnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer medium that is not acomputer readable storage medium and that can communicate, propagate, ortransport a program for use by or in connection with an instructionexecution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present embodiments may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects are described with reference to flowchart illustrations and/orblock diagrams of methods, apparatus (systems) and computer programproducts according to embodiments set forth herein. Each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, or portion of instructions,which comprises one or more executable instructions for implementing thespecified logical function(s). In some alternative implementations, thefunctions noted in the blocks may occur out of the order noted in theFigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts or carry out combinations of special purpose hardware and computerinstructions.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments. For example, thetechniques described herein are not dependent on whether color sensorsor black and white sensors are used, and may also be applied to Infrared(IR) light. There may also be embodiments where the different sensorsare placed in different cameras that capture the same scene, rather thanhaving all the sensors within a single camera. However, such a setup mayrequire more physical space for the cameras, and also require morecomplexity with respect to controlling the cameras. Thus, while such asetup is fully technically possible, it may be less feasible due tospace and cost requirements. Thus, many other variations that fallwithin the scope of the claims can be envisioned by those havingordinary skill in the art.

The terminology used herein was chosen to best explain the principles ofthe embodiments, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

The invention claimed is:
 1. A method for high dynamic range imaging,comprising: capturing first image data using a first image sensor and afirst exposure time, wherein the first image data is captured using arolling shutter such that different lines within the first image dataare captured at different first capture times; capturing two or moreinstances of second image data using one or more second image sensorsand one or more second exposure times that are shorter than the firstexposure time, wherein: the two or more instances of second image dataare captured using a rolling shutter such that different lines withineach instance of second image data are captured at different secondcapture times, the two or more instances of second image data overlap atleast in part with the first image data, a line of the first image datahas a spatially corresponding line in each instance of second imagedata, the spatially corresponding lines in the different instances ofsecond image data being captured at different second capture times; andfor a line in the first image data, selecting the spatiallycorresponding line from the instance of second image data whose secondcapture time is closest to the first capture time, to be merged with theline in the first image data to generate a high dynamic range image. 2.The method of claim 1, wherein the second exposure times are differentfor different instances of second image data.
 3. The method of claim 1,wherein the two or more instances of second image data is captured usingtwo or more second image sensors, wherein at least one of the secondexposure time and the second capture time differ between the two or moresecond image sensors.
 4. The method of claim 3, wherein instances ofsecond image data captured by the two or more image sensors overlap atleast in part.
 5. The method of claim 1, wherein the first and secondcapture times are determined with respect to one or more of: a reset ofa line and a read of a line.
 6. The method of claim 1, wherein the firstimage sensor and the one or more second image sensors have differentline times.
 7. The method of claim 3, wherein the second image sensorshave different line times.
 8. The method of claim 1, wherein selectingcomprises: for a line in the first image data, selecting the twospatially corresponding lines from the two instances of second imagedata whose second capture times are closest to the first capture time,wherein both selected lines are to be merged with the line in the firstimage data when generating a high dynamic range image.
 9. The method ofclaim 8, further comprising: assigning a weight to each selected lineamong the selected two spatially corresponding lines, to be used in thegeneration of the high dynamic range image.
 10. The method of claim 1,wherein at least one of: the first exposure times, the first capturetimes, the second exposure times, and the second capture times areselected in response to a user input.
 11. The method of claim 1, furthercomprising: for each line in the first image data, selecting thespatially corresponding line from the instance of second image datawhose second capture time is closest to the first capture time, to bemerged with the line in the first image data to generate a high dynamicrange image.
 12. The method of claim 1, further comprising: receivinginput regarding a region of interest in the first image data, wherein,for each line in the first image data intersecting the region ofinterest, a spatially corresponding line from a same instance of secondimage data is selected.
 13. A system for high dynamic range imaging,comprising: a memory; and a processor, wherein the memory containsinstructions that when executed by the processor causes the processor toperform a method that includes: capturing first image data using a firstimage sensor and a first exposure time, wherein the first image data iscaptured using a rolling shutter such that different lines within thefirst image data are captured at different first capture times;capturing two or more instances of second image data using one or moresecond image sensors and one or more second exposure times that areshorter than the first exposure time, wherein: the two or more instancesof second image data are captured using a rolling shutter such thatdifferent lines within each instance of second image data are capturedat different second capture times, the two or more instances of secondimage data overlap at least in part with the first image data, a line ofthe first image data has a spatially corresponding line in each instanceof second image data, the spatially corresponding lines in the differentinstances of second image data being captured at different secondcapture times; and for a line in the first image data, selecting thespatially corresponding line from the instance of second image datawhose second capture time is closest to the first capture time, to bemerged with the line in the first image data to generate a high dynamicrange image.
 14. A camera including a system for high dynamic rangeimaging, comprising: a memory; and a processor, wherein the memorycontains instructions that when executed by the processor causes theprocessor to perform a method that includes: capturing first image datausing a first image sensor and a first exposure time, wherein the firstimage data is captured using a rolling shutter such that different lineswithin the first image data are captured at different first capturetimes; capturing two or more instances of second image data using one ormore second image sensors and one or more second exposure times that areshorter than the first exposure time, wherein: the two or more instancesof second image data are captured using a rolling shutter such thatdifferent lines within each instance of second image data are capturedat different second capture times, the two or more instances of secondimage data overlap at least in part with the first image data, a line ofthe first image data has a spatially corresponding line in each instanceof second image data, the spatially corresponding lines in the differentinstances of second image data being captured at different secondcapture times; and for a line in the first image data, selecting thespatially corresponding line from the instance of second image datawhose second capture time is closest to the first capture time, to bemerged with the line in the first image data to generate a high dynamicrange image.
 15. A non-transitory computer readable storage mediumhaving program instructions embodied therewith, the program instructionsbeing executable by a processor to perform a method comprising:capturing first image data using a first image sensor and a firstexposure time, wherein the first image data is captured using a rollingshutter such that different lines within the first image data arecaptured at different first capture times; capturing two or moreinstances of second image data using one or more second image sensorsand one or more second exposure times that are shorter than the firstexposure time, wherein: the two or more instances of second image dataare captured using a rolling shutter such that different lines withineach instance of second image data are captured at different secondcapture times, the two or more instances of second image data overlap atleast in part with the first image data, a line of the first image datahas a spatially corresponding line in each instance of second imagedata, the spatially corresponding lines in the different instances ofsecond image data being captured at different second capture times; andfor a line in the first image data, selecting the spatiallycorresponding line from the instance of second image data whose secondcapture time is closest to the first capture time, to be merged with theline in the first image data to generate a high dynamic range image.